CLAIMS 
We claim: 

1 . A business method comprising the steps of: 

monitoring one or more multimedia items accessed by a user, each multimedia item having two 
5 or more disparate modalities, the disparate modalities being at least one or more visual 
modalities and one or more textual modalities; 

creating a visual feature vector for each of the visual modalities and a textual feature vector for 
each of the textual modalities; 

concatenating the visual feature vectors and the textual feature vectors into a unified feature 
1 0 vector; 

categorizing each of the multimedia items by categorizing the respective unified feature vector; 
and 

assembling a user profile based on the categorization. 

2. A business method, as in claim 1, further comprising the step of: 

1 5 using the user profile to match one or more multimedia items stored in one or more databases. 

3. A business method, as in claim 2, where one or more of the databases are part of a computer 
that is connected to one or more networks. 

4. A business method, as in claim 3, where the networks include any one or more of the 
following: an Internet, an intranet, an extranet, a corporate network, a government network, a 

20 infrared network, and a radio frequency network. 
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5. A business method, as in claim 1, where the categories include any one or more of the 
following: a product, a service, an interest, a retail item, a hobby, a food item, an item of 
clothing, a travel package, a vacation destination, a financial product, a business partner, a 
business interest, a medical product, a commercial, and a social interest, 

5 6. A business method, as in claim 1, where the category includes any one or more of the 
following services: consulting, legal, real estate, medical, technical, physical training, diet, 
cosmetic, fashion, governmental, automotive, design, architecture, personal assistants, games, 
on-line games of chance, dating services, and landscaping. 

7. A business method comprising the steps of: 
10 scanning one or more multimedia items in a database, each multimedia item having two or more 
disparate modalities, the disparate modalities being at least one or more visual modalities and 
one or more textual modalities; 

creating a visual feature vector for each of the visual modalities and a textual feature vector for 
each of the textual modalities; 

15 concatenating the visual feature vectors and the textual feature vectors into a unified feature 
vector; 

categorizing each of the multimedia items by categorizing the respective unified feature vector; 
and 



creating one or more indices of the database based on the categorization. 
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8. A business method, as in claim 7, where the database resides on any one or more of the 
following: a network server, a web site, a personal computer, a server farm, and a network disk 
array. 

9. A business method, as in claim 7, further comprising the step of: 
5 making a business decision based on the classifications. 

10. A business method, as in claim 9, where the categorization is used to organize a collection of 
the multimedia items into a database where the multimedia items are retrievable based on an 
annotation of the classification. 

1 1. A business method, as in claim 10, where the multimedia items that are retrieved are used as 
1 0 a response to a query of a search engine. 

12. A business method, as in claim 9, where the multimedia item is a multimedia e-mail and the 
multimedia e-mail is routed based on one or more of the categories assigned to the multimedia 
e-mail. 

13. A business method, as in claim 12, where the multimedia e-mail is routed to any one or more 
15 of the following: a sender, a folder, a person, a personal folder, a corporate folder, and a 

corporate department. 

14. A business method, as in claim 13, where the multimedia e-mail is multiplied before being 
routed. 

15. A business method comprising the steps of: 
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scanning one or more multimedia items in a database, each multimedia item having two or more 
disparate modalities, the disparate modalities being at least one or more visual modalities and 
one or more textual modalities; 

creating a visual feature vector for each of the visual modalities and a textual feature vector for 
5 each of the textual modalities; 

concatenating the visual feature vectors and the textual feature vectors into a unified feature 
vector; 

categorizing each of the multimedia items by categorizing the respective unified feature vector; 
comparing one or more of the unified feature vectors to one or more other feature vectors; 
10 and 

making a decision based on the comparison. 

16. A business method, as in claim 15, where the multimedia items are digital multimedia copies 
and the feature vectors represent restricted multimedia items and the comparison determines if 
the digital multimedia copies are similar to the restricted multimedia items and the decision is 

1 5 that the digital multimedia copies are subject to a second restriction. 

17. A business method, as in claim 16, where the second restriction includes any one or more of 
the following: a copyright restriction, a trademark restriction, intellectual property restriction, 
parental guidance restriction, common decency restrictions, user-defined restrictions. 
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18. A business method, as in claim 16, where the digital multimedia copies are accessible over 
the Internet. 

19. A business method, as in claim 15, where the comparison indicates a degree of similarity and 
the decision is that the multimedia item contains a known content. 
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